Ingress ecmp in virtual distributed routing environment

ABSTRACT

A logical routing element (LRE) having multiple designated instances for routing packets from physical hosts (PH) to a logical network is provided. A PH in a network segment with multiple designated instances can choose among the multiple designated instances for sending network traffic to other network nodes in the logical network according to a load balancing algorithm. Each logical interface (LIF) of an LRE is defined to be addressable by multiple identifiers or addresses, and each LIF identifier or address is assigned to a different designated instance.

BACKGROUND

In a network virtualization environment, one of the more commonapplications deployed on hypervisors are 3-tier apps, in which aweb-tier, a database-tier, and app-tier are on different L3 subnets.This requires IP (internet protocol) packets traversing from one virtualmachine (VM) in one subnet to another VM in another subnet to firstarrive at a L3 router, then forwarded to the destination VM using L2 MAC(media access control) address. This is true even if the destination VMis hosted on the same host machine as the originating VM. This generatesunnecessary network traffic and causes higher latency and lowerthroughput, which significantly degrades the performance of theapplication running on the hypervisors. Generally speaking, thisperformance degradation occurs whenever any two VMs in two differentnetwork segments (e.g., different IP subnet, different L2 segments, ordifferent overlay logical networks) communicate with each other.

FIG. 1 illustrates a logical network implemented over a networkvirtualization infrastructure 100, in which virtual machines (VMs)121-129 belonging to different network segments communicate with eachother through a shared L3 router 110. The VMs 121-129 are running onhost machines 131-133, which are physical machines communicativelylinked with each other and to the shared L3 router 110 by a physicalnetwork 105. A VM in network segment A can only communicates with a VMin network segment B through the physical network 105 and the shared L3router 110, whether the VMs are in different host machines (e.g., fromthe VM 122 to the VM 127) or in the same host machine (e.g., from the VM125 to the VM 126).

U.S. patent application Ser. No. 14/137,862, filed on Dec. 20, 2013,describes a logical router element (LRE) that operates distributivelyacross different host machines as a virtual distributed router (VDR).Each host machine operates its own local instance of the LRE as amanaged physical routing element (MPRE) for performing L3 packetforwarding for the VMs running on that host. The LRE therefore makes itpossible to forward data packets locally (i.e., at the originatinghypervisor) without going through a shared L3 router.

Furthermore, an LRE as described by U.S. patent application Ser. No.14/137,862 not only performs L3 routing for VMs operating in hostmachines that operate the LRE, but also performs L3 routing for physicalrouters/hosts or other network nodes that do not operate the LRE. Oneparticular host machine operating the LRE is selected as the designatedhost machine, and its MPRE is the designated instance of the LRE forhandling L3 routing of traffic from the physical routers.

SUMMARY

In some embodiments, a logical routing element (LRE) includes one ormore logical interfaces (LIFs) that each serve as an interface to acorresponding segment of a logical network. Each network segment has itsown logical interface to the LRE, and each LRE has its own set oflogical interfaces. In some embodiments, at least one of the LIFs of aLRE is defined to be addressable by two or more identifiers (e.g., IPaddresses). Some embodiments allow each LIF identifier to serve as adestination address for network traffic. In some embodiments, a networksegments can encompass multiple IP subnets, and a LIF interfacing such anetwork segment is addressable by IP addresses that are in different IPsubnets. In some embodiments, a network segment that is an overlayencapsulation network (e.g., VXLAN or VLAN) includes multiple IPsubnets.

A physical host (PH) is a network node that belongs to a logical networkbut does not operate a local instance of the logical network's LRE. Insome embodiments, network traffic from a PH to a VM is routed by adesignated host machine that does operate a local instance of the LRE(i.e., MPRE). The local instance of the LRE running on such a designatedhost is referred as a “designated instance” or “DI” in some embodiments.In some embodiments, a logical network (or an LRE) has multipledesignated instances for some or all of the network segments. A PH in anetwork segment with multiple designated instances can choose among themultiple designated instances for sending network traffic to othernetwork nodes in the logical network for load balancing purposes. Inorder to support multiple designated instances per network segment, acorresponding LIF in some embodiments is defined to be addressable bymultiple identifiers or addresses (e.g., IP addresses), where each LIFidentifier or address is assigned to a different designated instance. Insome embodiments, each LIF identifier serves as a destination addressfor network traffic. Each designated instance (DI) assigned to aparticular LIF identifier in turn handles network traffic for thatparticular assigned LIF identifier.

Some embodiments advertise the IP addresses of the LIF of thatparticular network segment as a list of available next hops. Once a listof designated instances is made available to a physical host, thephysical host is able to select any one of the designated instances as anext hop into the logical network. Such selection can be based on anynumber of criteria and can be made for any number of purposes. In someembodiments, a physical host selects a designated instance as the nexthop based on current network traffic information in order to balance thetraffic load between the different designated host machines. In someembodiments, a PH uses the list of designated instances to perform ECMP(Equal Cost Multi-path Routing) algorithms on ingress network traffic tothe logical network.

In some embodiments, packets coming from physical hosts (PHs) rely onrouting table entries in designated instances for routing. In someembodiments, these entries are filled by address resolution protocols(ARP) initiated by PHs or by DIs themselves. In some embodiments, a PHthat has received a list of IP addresses as next hops performs ARPoperation to translate the received L3 IP address into L2 MAC addressesin order to ascertain the PMAC addresses of the designated instances. Insome embodiments, the designated instances not only resolve IP addressesfor packets that come from external PHs, but also for packets comingfrom VMs running on host machines having a local instance of the LRE.The routing utilizes routing table entries in the available designatedinstances of a particular LIF.

In some embodiments, each MPRE select a designated instance forrequesting address resolution based on the destination IP address. Suchaddress resolution requests and address resolution replies are UDPmessages in some embodiments. In some embodiments, an MPRE would makesuch an address resolution request to a designated instance that isassociated with a LIF address that is in a same IP subnet as thedestination IP address. In some embodiments, each designated instance isfor resolving IP addresses that are in the same subnet as its assignedLIF IP address. In some embodiments, when a designated instance is notable to resolve a destination IP address upon receiving an addressresolution request, it will perform an ARP operation in order to resolvethe unknown IP address.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawings, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a logical network implemented over a networkvirtualization infrastructure, in which virtual machines (VMs) ondifferent segments or subnets communicate through a shared router.

FIG. 2 conceptually illustrates a virtualized network environment thatuses LREs to implement L3 packet forwarding between network nodes.

FIG. 3 illustrates LIFs that interface network segments that include oneor more IP subnets.

FIG. 4 illustrates the physical implementation of LREs in host machinesof a network virtualization infrastructure.

FIG. 5 illustrates a host machine running a virtualization software thatincludes a MPRE of an LRE.

FIG. 6 illustrates the use of MPREs for performing distributed L3routing for VMs in different host machines.

FIG. 7 illustrates the distributed L3 routing of data packets from theVMs to a PH.

FIG. 8 conceptually illustrates multiple designated instances for a LIFin a logical network.

FIG. 9 illustrates L3 routing of packets from a PH to VMs in the logicalnetwork by using two different designated instances.

FIG. 10 illustrates conceptually illustrates a LRE in which each LIF hasmultiple IP addresses, and each IP address has its own correspondingdesignated instance.

FIG. 11 conceptually illustrates a network virtualization infrastructurehaving host machines that implement a logical network based on the LREof FIG. 10.

FIG. 12 conceptually illustrates the advertising of LIF IP addresses asa list of next hops to physical hosts in the network virtualizationinfrastructure.

FIG. 13 illustrates a network system in which routers for ingressnetwork traffic into a logical network perform ECMP based on lists ofadvertised available next-hops.

FIG. 14 conceptually illustrates a process performed by a physical hostfor selecting a designated instance of an LRE for routing.

FIG. 15 conceptually illustrates a process for providing multipledesignated instances to external physical host machines.

FIG. 16 illustrates ARP operations for resolving LIF IP addressesadvertised to the PHs.

FIG. 17 a-b illustrates multiple designated instances acting as ARPproxies when they receive data packets with unknown destination IPaddresses from a PH.

FIG. 18 illustrates a designated instance using its existing routingtable entry to route a data packet from a PH without initiating an ARPoperation.

FIG. 19 illustrates the routing of a packet from a VM in a host machineoperating a MPRE to a physical host that is not operating a MPRE.

FIG. 20 illustrates an ARP operation performed by a designated instancewhen it is unable to resolve an IP address upon receiving an addressresolution request.

FIG. 21 conceptually illustrates a process for processing a data packetat an MPRE.

FIG. 22 conceptually illustrates a process for performing addressresolution at a designated instance MPRE.

FIG. 23 conceptually illustrates a process for performing packet routingand forwarding at an MPRE in some embodiments.

FIG. 24 illustrates a network virtualization infrastructure, in whichlogical network specifications are converted into configurations forLREs in host machines.

FIG. 25 conceptually illustrates the delivery of configuration data froma network manager to LREs operating in individual host machines.

FIG. 26 illustrates a structure of the configuration data sets that aredelivered to individual host machines.

FIG. 27 illustrates the gathering and the delivery of dynamic routinginformation for LREs.

FIG. 28 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following description, numerous details are set forth for thepurpose of explanation. However, one of ordinary skill in the art willrealize that the invention may be practiced without the use of thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form in order not to obscure the descriptionof the invention with unnecessary detail.

In some embodiments, a logical routing element (LRE) includes one ormore logical interfaces (LIFs) that each serve as an interface to acorresponding segment of the network. Each network segment has its ownlogical interface to the LRE, and each LRE has its own set of logicalinterfaces. In some embodiments, at least one of the LIFs of a LRE isdefined to be addressable by two or more identifiers (e.g., IPaddresses). Some embodiments allow each LIF identifier to serve as adestination address for network traffic. In some embodiments, a networksegments can encompass multiple IP subnets, and a LIF interfacing such anetwork segment is addressable by IP addresses that are in different IPsubnets. In some embodiments, a network segment that is an overlayencapsulation network (e.g., VXLAN or VLAN) includes multiple IPsubnets.

For some embodiments, FIG. 2 conceptually illustrates a virtualizednetwork environment 200 that uses LREs to implement L3 packet forwardingbetween network nodes. As illustrated, the virtualized networkenvironment 200 is a multi-tenancy network environment that serves twodifferent tenants X and Y, and is implementing two different logicalnetworks 201 and 202 for these two different tenants. The logicalnetwork 201 includes segments A, B, C, and D for tenant X, while thelogical network 202 includes network segments E, F, G, and H for tenantY. Each segment includes one or more network nodes that are each labeledeither as “VM” (virtual machine) or “PH” (physical host). The logicalnetwork 201 has a LRE 211 for handling L3 routing between networksegments A, B, C, and D, while the logical network 202 has a LRE 212 forhandling L3 routing between network segments E, F, G, and H.

In some embodiments, the virtualized network environment 200 isimplementing the logical networks 201 and 202 over a virtualizationinfrastructure that includes several host machines interconnected by aphysical network, as described in more detail below. Some of these hostmachines are operating virtualization software or hypervisors that allowthem to host one or more VMs. Some of these host machines are alsooperating local instances of the LREs as managed physical routingelements (MPREs) that allow the host machines to distributively performL3 routing between network nodes in different network segments. EachMPRE (i.e., a local instance of an LRE) running on a host machinefunctions as the local physical router for the VMs operating on thathost machine. Logical routing elements (LRE) or virtual distributedrouters (VDR) are described in U.S. patent application Ser. No.14/137,862, which is hereby incorporated by reference.

Each network segment includes one or more individually addressablenetwork nodes that consumes, generates, or forwards network traffic. Insome embodiments, a network segment is a portion of the network (e.g.,an IP subnet). In some embodiments, a network segment is defined by a L2logical switch and includes network nodes interconnected by that logicalswitch. In some embodiments, a network segment is an encapsulationoverlay network such as VXLAN or VLAN. Such a network segment can spanmultiple data centers and/or include multiple IP subnets in someembodiments. In some embodiments, a logical network can includedifferent types of network segments (e.g., a mixture of VLANs andVXLANs). In some embodiments, network nodes in a same segment are ableto communicate with each other by using link layer (L2) protocols (e.g.,according to each network node's L2 MAC address), while network nodes indifferent segments of the network cannot communicate with each otherwith a link layer protocol and must communicate with each other throughnetwork layer (L3) routers or gateways.

As illustrated in FIG. 2, some of these network nodes are virtualmachines (“VM”) running on host machines, while others are stand-alonenetwork nodes such as physical routers or physical host machines (“PH”).For some embodiments, a VM is a network node that is hosted by a hostmachine. Each of the host machines also operates a local instance of aLRE as its MPRE such that packets from the VMs can be routed locally atthat host machine to other segments of its logical network. Conversely,a PH is a network node that is not hosted by such a host machine. A PHdoes not have a local instance of a LRE to locally route its packet toother segments of its logical network. In some embodiments, a PHbelonging to a network segment uses a MPRE of another host machine(i.e., an LRE instance local to another host machine) for routing withinthe logical network. Routing for PH network nodes will be furtherdescribed in Section II below.

The LREs 211 and 212 are the logical routers for the logical networks201 and 202, respectively. The LRE 211 handles routing only for thetraffic of tenant X while the LRE 212 handles routing only for thetraffic of tenant Y. Consequently, the network traffic of tenant X isentirely isolated in the logical plane from the network traffic oftenant Y, although they may share physical resources, as furtherdescribed below.

As mentioned, an LRE operates distributively across the host machines inits logical network as a virtual distributed router (VDR), where eachhost machine operates its own local instance of the LRE as a MPRE forperforming L3 packet forwarding for the VMs running on that host. InFIG. 2, the LRE 211 (LRE for tenant X) is illustrated as encompassingMPREs 221-223, while the LRE 212 (LRE for tenant Y) is illustrated asencompassing MPREs 231-233. In other words, each of the MPREs 221-223 isa local instance of the LRE 211 running on a different host machine fortenant X, while each of MPRE 231-233 is a local instance of the LRE 212running on a different host machine for tenant Y.

As illustrated, each of LREs 211 and 212 includes a set of logicalinterfaces (LIFs) that each serves as an interface to a particularsegment of the network. The LRE 211 has LIF A, LIF B, LIF C, and LIF Dfor handling packets to and from the network segments A, B, C, and D,respectively, while the LRE 212 has LIF E, LIF F, LIF G, and LIF G forhandling packets to and from the network segments E, F, G, and H,respectively. Each logical interface is assigned its own set ofidentifiers (e.g., IP address or overlay network identifier) that isunique within the network virtualization environment 200. For example,LIF A of LRE 211 assigned IP addresses 1.1.1.251, 1.1.1.252, and1.1.1.253, and LIF F is assigned IP addresses 4.1.2.251, 4.11.2.252, and4.11.2.253. Each of these LIF identifiers can serve as a destinationaddress for network traffic, in other words, the multiple IP addresses(or identifiers) of a LIF allows the LIF to appear as multiple differentnetwork traffic destinations. For example, in some embodiments, each LIFIP address serves as an address of a default gateway or ARP proxy fornetwork nodes of its particular network segment. Having multiple IPaddresses per LIF provides the network nodes in the correspondingnetwork segments a list of gateways or proxies to choose from.

In some embodiments, a network segments can encompass multiple IPsubnets, and a LIF interfacing such a network segment is addressable byIP addresses that are in different IP subnets. In some embodiments, anetwork segment that is an overlay encapsulation network (e.g., VXLAN orVLAN) includes multiple IP subnets. FIG. 3 illustrates LIFs thatinterface network segments that include one or more IP subnets.Specifically, FIG. 3 illustrates LIFs A-H of the LREs 211 and 212 andtheir corresponding network segments A-H.

As illustrated, some of the network segments (e.g., network segments Aand E) include only one IP subnet. A LIF interfacing such a networksegment have all of its LIF addresses in one IP subnet. For example, thenetwork segment A only includes network nodes in IP subnet 1.1.1.x, andthe LIF addresses for its corresponding LIF (LIF A) are also all in theIP subnet 1.1.1.x (i.e., 1.1.1.251, 1.1.1.252, 1.1.1.253). On the otherhand, some of the network segments include multiple IP subnets. Forexample, the network segment B includes IP subnets 1.1.2.x and 1.1.12.x,while the segment C includes IP subnets 1.1.3.x, 1.1.13.x, and 1.1.23.x.In some embodiments, a LIF of a network segment also has LIF IPaddresses in those multiple subnets of the network segments. Forexample, LIF B has IP addresses in IP subnet 1.1.2.x (1.1.2.251) as wellas in IP subnet 1.1.12.x (1.1.12.252 and 1.1.12.253). In some of theseembodiments, network nodes in a particular IP subnet uses only LIFaddresses in the same IP subnet when accessing the LIF. For example, insome embodiments, VMs in subnet 1.1.14.x of segment D uses only theaddresses 1.1.14.252 or 1.1.14.253 to address LIF D but not 1.1.4.251,even though 1.1.4.251 is also an address of the same LIF.

In some embodiments, the IP addresses of a LIF need not correspondexactly with the IP subnets in the LIF's network segment. For example, aLIF may have an IP address that is not in any of the network segment'ssubnets (e.g., the network segment E does not have IP subnet thatencompasses the LIF address 4.10.1.253 in LIF E), or a LIF may have asubnet that does not have at least one LIF address that is in thatsubnet (e.g., LIF H does not have a LIF address in the subnet 4.1.14.x).

Several figures below (e.g., FIGS. 6-9) use the IP address and networksegment assignment of FIGS. 2-4. One of ordinary skill would understandthat the values of IP addresses and labels of network segments of FIGS.2-9 are arbitrarily chosen for purposes of illustration, and that thevarious embodiments described in those figures as well as other figuresare entirely independent of the specific names or numerical valueschosen.

Several more detailed embodiments of the invention are described below.Section I describes distributed routing using LREs in virtualizednetwork environment. Section II describes various applications of a LIFthat has multiple LIF identifiers. Section III describes the control andconfiguration of LRE. Finally, section IV describes an electronic systemwith which some embodiments of the invention are implemented.

I. Logical Routing Element

As mentioned, some embodiments use logical routing elements (LREs) forrouting packets between network nodes in different network segments.These LREs operate in a distributed manner across multiple hostmachines, each of these host machines operating a local instance of theLRE as its managed physical routing element (MPRE). In some embodiments,each of these host machines is also operating a virtualization softwareor a hypervisor that allows it to host one or more virtual machines(VMs) and to provide network access to those VMs. In some embodiments,the host machines running the LREs are in a network virtualizationinfrastructure over a physical network. Such a network virtualizationinfrastructure in some embodiments includes physical network nodes (suchas external edge routers) that belong to a network segment that isserved by one of the LREs and yet does not operate the LRE itself.

FIG. 4 illustrates the physical implementation of the LREs 211 and 212in host machines of a network virtualization infrastructure 400.Specifically, the figure illustrates the (partial) implementation of thelogical networks 201 and 202 in host machines 401-403. The host machines401-403 are communicatively interconnected through a physical network490. Physical hosts (PH) 491-494 are also connected to the physicalnetwork 490 and communicatively interconnected with host machines401-403.

As illustrated, the host machine 401 is hosting VMs 411-416, the hostmachine 402 is hosting VMs 421-426, and the host machine 403 is hostingVMs 431-436. These VMs belong to different network segments. Namely, theVM 411 belongs to segment A, the VM 412 belong to segment B, the VM 413,421, 422 belong to segment C, the VMs 431, 432 belong to segment D, theVMs 414, 424 belong to segment E, the VMs 425 and 433 belong to segmentF, the VMs 415, 416 belong to segment G, and the VMs 426, 434-436 belongto segment H.

Each host machine is operating two MPREs for the two different LREs 211and 212. Specifically, the host machine 401 is operating MPREs 441 and451, the host machine 402 is operating MPREs 442 and 452, and the hostmachine 403 is operating MPREs 443 and 453. The MPREs 441-443 are localinstances of the LRE 211 operating in the host machines 401-403,respectively, for the logical network 201 of tenant X. The MPREs 451-453are local instances of the LRE 212 operating in the host machines401-403, respectively, for the logical network 202 of tenant Y.

A MPRE residing on a host machine has a set of LIFs (i.e., the LIFs ofthe LRE) for interfacing with the VMs operating on that host machine.For example, the MPRE 441 has LIFs A, B, C, and D as the local instanceof the LRE 211. The LIF A of the MPRE 441 serves the VM 411 (a segment AVM), the LIF B of MPRE 441 serves the VM 412 (a segment B VM), and theLIF C of MPRE 441 serves the VM 413 (a segment C VM). As illustrated, anMPRE of a LRE/logical network may reside on a host machine that does nothave VMs in all network segments, and the MPRE therefore may have LIFsthat are inactive. For example, the host machine 401 does not have a VMbelonging to segment D, and the LIF D of its MPRE 441 is therefore notactivated (illustrated with dashed borders).

Each MPRE of a host machine handles the L3 routing of packets comingfrom the VMs that are served by the MPRE's LIFs. In other words, eachMPRE handles the L3 routing of the VMs belonging to network segmentsthat form the logical network of its parent LRE. For example, the MPRE441 performs L3 routing for VMs 411-413 (belonging to network segmentsA, B, C of the logical network 201), while the MRPE 442 performs L3routing for VMs 414-416 (belonging to network segments E and G of thelogical network 202).

Each host machine is also operating a managed physical switching element(MPSE) for performing L2 level switching between the VMs and the MPREson that host machine. The MPSE of each host machine also has an uplinkconnection to the physical network 490 so the VMs and the MPREs in thehost machine can exchange packets with network nodes outside of the hostmachine (e.g., VMs in other host machines and PHs) over the physicalnetwork 490. For example, packets can arrive at the MPSE 461 of the host401 from the physical network 490 through the uplink, from one of theMPREs (441 or 442), or from one of the VMs (411-416). Packets thatrequire L3 level routing are forwarded by the MPSE 461 to one of theMPREs 441 or 451, and the routed packet are sent back to the MPSE 461 tobe forwarded to their L2 destination within the host machine 401 oroutside of the host machine reachable by the physical network 490.

In some embodiments, all MPREs are addressable within its host machine(i.e., by the MPSE of the host machine) by a same virtual MAC address(VMAC), while each MPRE is addressable from network nodes outside of itshost machine by a physical MAC address (PMAC) that uniquely identifiesthe MPRE. Such a PMAC in some embodiments distinguishes a MPRE operatingin one host machine from another MPRE operating in another host machine,even when those MPREs are instances of a same LRE. In some embodiments,though MPREs of different tenants on a same host machine are addressableby a same MAC (either VMAC or PMAC) at the MPSE of the host machine, theMPREs are able to keeps packets of different logical networks (and ofdifferent clients) separate by using network segment identifiers (e.g.,VNI, VXLAN ID or VLAN tag or ID). For example, the LIFs A, B, C, and Dof MPRE 441 ensures that the MPRE 441 receives only packets withidentifiers for network segments A, B, C, or D, while the LIFs E, F, G,and H of MPRE 442 ensures that the MPRE 442 receives only packets withidentifiers for network segments E, F, G, and H. The operations of MPSEare described in U.S. patent application Ser. No. 14/137,862.

Physical hosts (PH) 491-494 are network nodes that, though belonging tological networks 201 or 202, do not operate a local instance of eitherthe LRE 211 or the LRE 212. Specifically, the PH 491 belongs to networksegment A, the PHs 492 and 493 belong to network segment C, and the PH493 belong to network segment G. In some embodiments, a PH is a physicalhost machine that does not run virtualization software at all and doesnot host any VMs. In some embodiments, some physical host machines arelegacy network elements (such as filer or another non-hypervisor/non-VMnetwork stack) built into the underlying physical network, which used torely on standalone routers for L3 layer routing. In some embodiments, aPH is an edge router or a routing gateway that serves as an interfacefor the logical networks 201 or 202 with other external networks. Insome embodiments, such an edge router is a VM running on a host machinethat operates hypervisor/virtualization software, but the host machineof the edge router does not operate an LRE for either logical network201 or 202. In order to perform L3 layer routing for these PH networknodes, some embodiments designate one or more MPREs running in the hostmachines of the network virtualization infrastructure 400 to act as adedicated routing agent (designated instance or designated MPRE) forthese PHs. In some embodiments, L2 traffic to and from these PHs arehandled by local instances of MPSEs in the host machines without havingto go through a designated MPRE. Designated instances will be furtherdescribed in Section II.a below.

In some embodiments, a LRE operates within a virtualization software(e.g., a hypervisor, virtual machine monitor, etc.) that runs on a hostmachine that hosts one or more VMs (e.g., within a multi-tenant datacenter). The virtualization software manages the operations of the VMsas well as their access to the physical resources and the networkresources of the host machine, and the local instantiation of the LREoperates in the host machine as its local MPRE. For some embodiments,FIG. 5 illustrates a host machine 500 running a virtualization software505 that includes a MPRE of an LRE. The host machine connects to, e.g.,other similar host machines, through a physical network 590. Thisphysical network 590 may include various physical switches and routers,in some embodiments.

As illustrated, the host machine 500 has access to a physical network590 through a physical NIC (PNIC) 595. The host machine 500 also runsthe virtualization software 505 and hosts VMs 511-514. Thevirtualization software 505 serves as the interface between the hostedVMs and the physical NIC 595 (as well as other physical resources, suchas processors and memory). Each of the VMs includes a virtual NIC (VNIC)for accessing the network through the virtualization software 505. EachVNIC in a VM is responsible for exchanging packets between the VM andthe virtualization software 505. In some embodiments, the VNICs aresoftware abstractions of physical NICs implemented by virtual NICemulators.

The virtualization software 505 manages the operations of the VMs511-514, and includes several components for managing the access of theVMs to the physical network (by implementing the logical networks towhich the VMs connect, in some embodiments). As illustrated, thevirtualization software includes several components, including a MPSE520, a MPRE 530, a controller agent 540, a VTEP 550, and a set of uplinkpipelines 570.

The controller agent 540 receives control plane messages from acontroller or a cluster of controllers. In some embodiments, thesecontrol plane message includes configuration data for configuring thevarious components of the virtualization software (such as the MPSE 520and the MPRE 530) and/or the virtual machines. In the exampleillustrated in FIG. 5, the controller agent 540 receives control planemessages from the controller cluster 560 from the physical network 590and in turn provides the received configuration data to the MPRE 530through a control channel without going through the MPSE 520. However,in some embodiments, the controller agent 540 receives control planemessages from a direct data conduit (not illustrated) independent of thephysical network 590. In some other embodiments, the controller agentreceives control plane messages from the MPSE 520 and forwardsconfiguration data to the router 530 through the MPSE 520. Thecontroller agent and the configuration of the virtualization softwarewill be further described in Section III below.

The VTEP (VXLAN tunnel endpoint) 550 allows the host 500 to serve as atunnel endpoint for logical network traffic (e.g., VXLAN traffic). VXLANis an overlay network encapsulation protocol. An overlay network createdby VXLAN encapsulation is sometimes referred to as a VXLAN network, orsimply VXLAN. When a VM on the host 500 sends a data packet (e.g., anethernet frame) to another VM in the same VXLAN network but on adifferent host, the VTEP will encapsulate the data packet using theVXLAN network's VNI and network addresses of the VTEP, before sendingthe packet to the physical network. The packet is tunneled through thephysical network (i.e., the encapsulation renders the underlying packettransparent to the intervening network elements) to the destinationhost. The VTEP at the destination host decapsulates the packet andforwards only the original inner data packet to the destination VM. Insome embodiments, the VTEP module serves only as a controller interfacefor VXLAN encapsulation, while the encapsulation and decapsulation ofVXLAN packets is accomplished at the uplink module 570.

The MPSE 520 delivers network data to and from the physical NIC 595,which interfaces the physical network 590. The MPSE also includes anumber of virtual ports (vPorts) that communicatively interconnects thephysical NIC with the VMs 511-514, the MPRE 530 and the controller agent540. Each virtual port is associated with a unique L2 MAC address, insome embodiments. The MPSE performs L2 link layer packet forwardingbetween any two network elements that are connected to its virtualports. The MPSE also performs L2 link layer packet forwarding betweenany network element connected to any one of its virtual ports and areachable L2 network element on the physical network 590 (e.g., anotherVM running on another host). In some embodiments, a MPSE is a localinstantiation of a logical switching element (LSE) that operates acrossthe different host machines and can perform L2 packet switching betweenVMs on a same host machine or on different host machines.

The MPRE 530 performs L3 routing (e.g., by performing L3 IP address toL2 MAC address resolution) on data packets received from a virtual porton the MPSE 520. Each routed data packet is then sent back to the MPSE520 to be forwarded to its destination according to the resolved L2 MACaddress. This destination can be another VM connected to a virtual porton the MPSE 520, or a reachable L2 network element on the physicalnetwork 590 (e.g., another VM running on another host, a physicalnon-virtualized machine, etc.).

As mentioned, in some embodiments, a MPRE is a local instantiation of alogical routing element (LRE) that operates across the different hostmachines and can perform L3 packet forwarding between VMs on a same hostmachine or on different host machines. In some embodiments, a hostmachine may have multiple MPREs connected to a single MPSE, where eachMPRE in the host machine implements a different LRE. MPREs and MPSEs arereferred to as “physical” routing/switching element in order todistinguish from “logical” routing/switching elements, even though MPREsand MPSE are implemented in software in some embodiments. In someembodiments, a MPRE is referred to as a “software router” and a MPSE isreferred to a “software switch”. In some embodiments, LREs and LSEs arecollectively referred to as logical forwarding elements (LFEs), whileMPREs and MPSEs are collectively referred to as managed physicalforwarding elements (MPFEs).

In some embodiments, the MPRE 530 includes one or more logicalinterfaces (LIFs) that each serves as an interface to a particularsegment of the network. In some embodiments, each LIF is addressable byits own IP address and serve as a default gateway or ARP proxy fornetwork nodes (e.g., VMs) of its particular segment of the network. Asdescribed in detail below, in some embodiments, all of the MPREs in thedifferent host machines are addressable by a same “virtual” MAC address,while each MPRE is also assigned a “physical” MAC address in orderindicate in which host machine does the MPRE operate.

The uplink module 570 relays data between the MPSE 520 and the physicalNIC 595. The uplink module 570 includes an egress chain and an ingresschain that each performs a number of operations. Some of theseoperations are pre-processing and/or post-processing operations for theMPRE 530. The operations of the uplink module are described in U.S.patent application Ser. No. 14/137,862.

As illustrated by FIG. 5, the virtualization software 505 has multipleMPREs from multiple different LREs. In a multi-tenancy environment, ahost machine can operate virtual machines from multiple different usersor tenants (i.e., connected to different logical networks). In someembodiments, each user or tenant has a corresponding MPRE instantiationin the host for handling its L3 routing. In some embodiments, though thedifferent MPREs belong to different tenants, they all share a same vPorton the MPSE 520, and hence a same L2 MAC address (VMAC or PMAC). In someother embodiments, each different MPRE belonging to a different tenanthas its own port to the MPSE.

The MPSE 520 and the MPRE 530 make it possible for data packets to beforwarded amongst VMs 511-514 without being sent through the externalphysical network 590 (so long as the VMs connect to the same logicalnetwork, as different tenants' VMs will be isolated from each other).

A MPRE running on a host machine allows L3 routing of packets betweenVMs running on a same host machine to be done locally at the hostmachine without having to go through the physical network. FIG. 6illustrates the use of MPREs for performing distributed L3 routing forVMs in different host machines. Specifically, FIG. 6 illustrates MPREsperforming L3 routing between VMs in a same host machine and between VMsin different host machines.

As illustrated, a physical network 690 supports network communicationsbetween host machines 601-604 (the host machine 604 is illustrated inFIG. 7). The host machines 601-604 are operating MPREs 631-634,respectively. The MPREs 631-634 are local instances of a same LRE. EachMPRE has a corresponding routing table (641-644 for MPREs 631-634,respectively) for mapping L3 IP addresses into L2 MAC addresses. The LRE(and hence the MPREs) have LIFs for network segments A, B, C, and D. Thehost machine 602 is hosting a VM 612, which is a network node in networksegment B. The host machine 604 is hosting a VM 614, which is a networknode in network segment D. The host machine 601 is hosting VMs 611 and615, which are network nodes in network segments A and C, respectively.

FIG. 6 illustrates the routing of a packet 670 from the VM 612 to the VM614 in operations labeled ‘1’ through ‘5’. The VMs 612 and 614 are VMsoperating in different host machines. The packet 670 indicates that itis from IP address 1.1.2.2, and that it is destined for IP address1.1.4.4. At operation ‘1’, the VM 612 sends the packet 670 to the MPRE632 through its LIF B because the VM 612 is a network node in networksegment B (as indicated by its IP address 1.1.2.2). At operation ‘2’,the MPRE 632 uses its routing table 642 to map the destination IPaddress 1.1.4.4 to “MAC4”, which is the L2 address of the VM 614 in hostmachine 604.

At operation ‘3’, the MPRE realizes that destination address 1.1.4.4 isin a subnet in network segment D and therefore uses its LIF D to sendout the packet 670 with “MAC4” as the destination MAC address. Thoughnot illustrated, the packet 670 is forwarded out by an MPSE in the hostmachine 602. The MPSE recognizes that “MAC4” is not in the host machine602 and sends it out to the physical network 690.

At operation ‘4’, the packet 670 reaches the host machine 604. Since thepacket 670 is already routed (i.e., having a routed MAC address), theMPSE of the host machine 604 in operation ‘5’ forward the packet 670 toL2 address “MAC4” (i.e., the VM 614) without going through the MPRE 634.

FIG. 6 also illustrates the routing of a packet 680 from the VM 611 tothe VM 615 in operations labeled ‘6’ through ‘8’. The packet 680indicates that it is from IP address 1.1.1.1, and that it is destinedfor IP address 1.1.3.5. The VMs 611 and 615 are VMs operating in a samehost machine 601. At operation ‘6’, the VM 611 sends the packet 680 tothe MPRE 631 through its LIF A because the VM 611 is a network node innetwork segment A (as indicated by its IP address 1.1.1.1). At operation‘7’, the MPRE 631 uses its routing table 641 to map the destination IPaddress 1.1.3.5 to “MAC5”, which is the L2 address of the VM 615.

At operation ‘8’, the MPRE realizes that destination address 1.1.3.3 isin a subnet belonging to network segment C and therefore uses its LIF Cto send out the packet 680 with “MAC5” as the destination MAC address.Though not illustrated, the packet is forwarded by an MPSE in the hostmachine 601. The MPSE recognizes that “MAC5” is in the host machine 601so it forwards the packet 680 directly to the VM 615 without goingthrough the physical network 690.

As mentioned, a physical host (PH) is a network node that belongs to alogical network but does not operate a local instance of the logicalnetwork's LRE. In some embodiments, network traffic from a PH to a VM istherefore routed by a designated host machine that does operate a localinstance of the LRE (i.e., MPRE). However, in some embodiments, theconverse is not true. Namely, network traffic from VMs to a PH is alwaysrouted locally, in a distributed fashion, by each host machine's ownMPRE without relying on a designated host.

FIG. 7 illustrates the distributed L3 routing of data packets from theVMs 611-614 to a PH 695. The packets from the VMs 611-614 are eachrouted locally at the MPRE of its host machine, even though thedestination is a PH 695 that does not operate an instance of the LRE.The VMs 611-614 are being hosted by host machines 601-604, respectively.The host machines 611-614 are communicatively linked with each other bythe physical network 690. The physical network 690 also connects the PH695 with the host machines 601-604. The host machines 601-604 areoperating MPREs 631-634, respectively, which are local instances of aLRE of a particular logical network. The PH 695 is a network node thatbelongs to a same particular logical network as the VMs 611-614, but thePH 695 does not operate the LRE of the particular logical network.

FIG. 7 illustrates four operations labeled ‘1’ through ‘4’ thatcorrespond to four different packet forwarding operations from VMs611-614 to the PH 695. The host machines hosting these VMs perform theforwarding operations locally by using their own MPREs in each of thesefour operations. Operation ‘1’ illustrates the routing of a packet 671from the VM 611 to the PH 695. The VM 611 is hosted by the host machine601, which is operating the MPRE 631. The MPRE 631 receives the packet671 at its LIF A (since the VM 611 is a network node at network segmentA) and uses its routing table 641 to map the destination IP address1.1.2.10 to “MAC10”, which is the MAC address of the PH 695. The routedpacket goes out of LIF B of the MPRE 641 (since the destination IPaddress 1.1.2.10 indicates that the destination network node is innetwork segment B). The local MPSE of the host machine 601 then sendsthe routed packet 671 out to the physical network 690 and then to the PHbased on the routed MAC address “MAC10”. Operations ‘3’ and ‘4’ likewiseillustrate the L3 routing of packet 673 from the VM 613 to the PH 695and packet 674 from the VM 614 to the PH 695. Each of these routingoperations is performed by a local MPRE (633 and 634) in each VM'srespective host machine (603 and 604) by using each MPRE's own routingtable (643 and 644).

In operations ‘1’, ‘3’, and ‘4’, the MPREs are performing L3 routingoperations since the PH 695 is on a different network segment than theVMs 611, 613 and 614. (The IP address of the PH 695 is 1.1.2.10, whichmakes the PH 695 part of network segment B. The IP address of VM 611 is1.1.1.1, which is in network segment A. The IP address of VM 613 is1.1.3.3, which is in network segment C. The IP address of VM 614 is1.1.4.4, which is in network segment D.) Operation ‘2’, on the otherhand, illustrates the forwarding of a packet 672 to the PH 695 from a VM612 that is in the same segment B as the PH 695 (the VM 612 is at IPaddress 1.1.2.2, which is also in segment B). If the packet 672 hasalready specified the destination MAC address (i.e., MAC10), in someembodiments, the MPSE of the host machine 602 would directly forward thepacket to the PH 695 via the physical network 690 without routing. Ifthe destination MAC address is unknown, the MPRE 632 in some embodimentswould perform a bridging operation to map the destination IP address1.1.2.10 to the destination MAC address MAC10. MPREs performing bridgingoperations are described in U.S. patent application Ser. No. 14/137,862.

II. Multiple Addresses Per LIF

a. Designated Instances for LIF Addresses

As mentioned, a physical host (PH) is a network node that belongs to alogical network but does not operate a local instance of the logicalnetwork's LRE. In some embodiments, network traffic from a PH to a VM istherefore routed by a designated host machine that does operate a localinstance of the LRE (i.e., MPRE). The local instance of the LRE runningon such a designated host is referred as a “designated instance” or “DI”in some embodiments, because it is a designated MPRE instance used tohandle traffic from physical hosts that do not have their own MPREs.

In some embodiments, a logical network (or an LRE) has multipledesignated instances for some or all of the network segments. A PH in anetwork segments with multiple designated instances can choose among themultiple designated instances for sending network traffic to othernetwork nodes in the logical network, for say, load balancing purposes.In order to support multiple designated instances per network segment, acorresponding LIF in some embodiments is defined to be addressable bymultiple identifiers or addresses (e.g., IP addresses), where each LIFidentifier or address is assigned to a different designated instance. Insome embodiments, each LIF identifier serves as a destination addressfor network traffic. Each designated instance (DI) assigned to aparticular LIF identifier in turn handles network traffic for thatparticular assigned LIF identifier.

FIG. 8 conceptually illustrates multiple designated instances for a LIFin a logical network 800. The logical network 800 is implementing a LRE830 for network segments A, B, C, and, D, and the LRE 830 has LIF A, LIFB, LIF C, and LIF D for serving as interfaces for these four networksegments. The logical network 800 is implemented over a networkvirtualization infrastructure that includes host machines 801-807interconnected by a physical network 890 (shown in FIG. 9). Each ofthese host machines is running a local instance of the LRE 830 as itsMPRE (i.e., MPREs 831-837).

The logical network 800 also includes two PHs 880 and 881. The PHs donot run their own local instances of the LRE 830 and therefore rely ondesignated instances for L3 routing within the logical network 800. TheIP address of the PH 880 is 1.1.2.10 and the IP address of the PH 881 is1.1.2.11, which indicates that both PH 880 and the PH 881 are in thenetwork segment B and interfaces the LRE 830 by using LIF B.

In the example of FIG. 8, the LIF B has three IP addresses: 1.1.2.251,1.1.2.252, and 1.1.2.253. The logical network has three designated hostmachines (and three DIs) for these three LIF addresses: the MPRE 831running on the host machine 801 is the DI for the LIF address 1.1.2.251,the MPRE 832 running on the host machine 802 is the DI for the LIFaddress 1.1.2.252, and the MPRE 833 running on the host machine 803 isthe DI for the LIF address 1.1.2.253. The MPREs in host machines 804-807are not DIs for the LIF B (though not illustrated, they can be DIs forother LIFs). Thus, the host machines 801, 802, and 803 can all serve asdesignated host machines for performing L3 routing on packets from thePHs 880 and 881.

As mentioned earlier, each MPRE is addressable from network nodesoutside of its host machine by a physical MAC address (PMAC), whichuniquely identifies the MPRE from other MPREs in other host machines. Insome embodiments, the PHs use the PMAC of a designated instance as itsfirst hop L2 destination. In other words, to send a packet to be routedby a DI, a PH would first send the packet to the DI by using the DI'sPMAC address. In the example of FIG. 8, the DI in the host machine 801has PMAC address “PMAC100”, the DI in the host machine 802 has PMACaddress “PMAC200”, and the DI in the host machine 803 has PMAC address“PMAC300”.

FIG. 9 illustrates L3 routing of packets from the PH 880 to VMs in thelogical network 800 by using two different designated instances. The PH880 is sending a packet 971 destined for IP address 1.1.3.3 and alsoanother packet 972 destined for IP address 1.1.4.4. The PH 880 uses theDI 833 (i.e., the MPRE of the host machine 803) as the first hop for thepacket 971 and the DI 831 (i.e., the MPRE of the host machine 801) asthe first hop for the packet 972.

Operations labeled ‘1’ through ‘4’ illustrates the routing of the packet971. At operation ‘1’, the PH 880 sends the packet 971 on to thephysical network 890. The packet 971 specifies that it is destined forIP 1.1.3.3 while its first hop MAC address is “PMAC300”. At operation‘2’, the packet 971 reaches the MPRE 833 in the host 803 based on theMAC address “PMAC300”, which is the PMAC of the MPRE 833. The packetenters the MPRE 833 through LIF B since the PH 880 is in network segmentB (IP address 1.1.2.10). At operation ‘3’, the MPRE 833 uses its routingtable 843 to translates the destination IP address 1.1.3.3 todestination MAC address “MAC3”. At operation ‘4’, the MPSE (notillustrated) of the host machine 803 recognizes that “MAC3” is the MACaddress of a VM 933 running within the host machine 803. The MPSE thenforwards the packet 971 to the VM 933.

Operations labeled ‘5’ through ‘9’ illustrates the routing of the packet972. At operation ‘5’, the PH 880 sends the packet 972 on to thephysical network 890. The packet 972 specifies that it is destined forIP 1.1.4.4 while its first hop MAC address is “PMAC100”. At operation‘6’, the packet 972 reaches the MPRE 831 in the host 801 based on theMAC address “PMAC100”, which is the PMAC of MPRE 831. The packet entersthe MPRE 831 through its LIF B since the PH 880 is in network segment B(IP address 1.1.2.10). At operation ‘7’, the MPRE 831 uses its routingtable 841 to translates the destination IP address 1.1.4.4 todestination MAC address “MAC4”. At operation ‘8’, the MPSE of the hostmachine 801 realizes that “MAC4” is not an address for any network nodewithin the host machine 801 and forwards the routed packet 972 out ontothe physical network 890. At operation ‘9’, the routed packet 972 withdestination “MAC4” reaches the host machine 804, whose MPSE (notillustrated) recognize it as the L2 address of a VM 934 running on thathost machine. The MPSE of the host machine 804 then forwards the routedpacket 972 to the VM 934, whose IP address is 1.1.4.4.

In some embodiments, different LIFs of a LRE have different sets of IPaddresses, and each IP address of a LIF has a corresponding designatedinstance. FIG. 10 illustrates conceptually illustrates a LRE 1000 inwhich each LIF has multiple IP addresses, and each IP address has itsown corresponding designated instance. The LRE 1000 (LRE X) has fourLIFs 1011-1014 for four different network segments. The LIFs 1011 and1012 are for VLAN network segments (VLAN100 and VLAN200). The LIFs 1013and 1014 are for VXLAN network segments (VXLAN500 and VXLAN600).

As illustrated, each of LIFs 1011-1014 has multiple IP addresses, andeach IP address is associated with a host machine that is operating alocal instance of the LRE X (i.e., MPRE) as the designated instance forthat IP address. In some embodiments, each IP address of a LIF isassociated with a different host machine. As mentioned earlier, in someembodiments, a PMAC of a MPRE is an address that is used to uniquelyidentify one MPRE in one host machine from other MPREs in other hostmachines, therefore, IP addresses associated with different PMACaddresses indicates designated instances in different host machines. Forexample, the LIF 1012 has IP addresses 2.1.2.251, 2.1.2.252, and2.1.2.253. The LIF IP addresses 2.1.2.251 has a designated instance withPMAC address “11:11:11:11:12:01” or “PMAC4”, the LIF IP addresses2.1.2.252 has a designated instance with PMAC address“11:11:11:11:12:02” or “PMAC5”, and the LIF IP addresses 2.1.2.253 has adesignated instance with PMAC address “11:11:11:11:12:01” or “PMAC6”.The three IP addresses of the LIF 1012 are therefore assigned to MPREsin three different host machines.

In some embodiments, one host machine can serve as the designated hostmachine (and its MPRE as the designated instance) for multiple differentIP addresses from multiple different LIFs. For example, the PMAC address“PMAC1” corresponds to both IP address 2.1.1.251 of the LIF 1011 and IPaddress 2.1.3.251 of the LIF 1013, i.e., the MPRE having “PMAC1” isserving as the designated instance for both of these LIF IP addresses.Likewise, the PMAC address “PMAC6” corresponds to both IP address2.1.2.253 of the LIF 1012 and IP address 2.1.4.253 of the LIF 1014. Inother words, the MPRE having “PMAC1” is a designated instance (and itshost machine the designated host machine) for both VLAN100 and VXLAN500,while the MPRE having “PMAC6” is a designated instance for both VLAN200and VXLAN600.

FIG. 11 conceptually illustrates a network virtualization infrastructure1100 having host machines that implement a logical network based on theLRE 1000 of FIG. 10. The network virtualization infrastructure 1100includes a number of host machines, including host machines 1101-1111.These host machines are hosting the VMs 1120. Each host machine isoperating a local instance of the LRE 1000 as MPRE. Each host machine isalso associated with a PMAC so its MPRE is uniquely addressable withinthe LRE 1000.

The network virtualization infrastructure 1100 also includes PH1181-1188, which are not operating a local instance of the LRE 1000. ThePH 1181-1182 are in VLAN100, the PH 1183-1184 are in VLAN200, the PH1185-1186 are in VXLAN500, and the PH 1187-1188 are in VXLAN600.

Some of the host machines, namely, host machines 1101-1111, areoperating MPREs that serve as designated instances for handling trafficfrom the PHs 1181-1188. Specifically, the host machines 1101, 1102, and1103 are serving as designated host machines for VLAN100 for handingtraffic from PHs 1181 and 1182, the host machines 1104, 1105, and 1106are serving as designated host machines for VLAN200 for handing trafficfrom PHs 1183 and 1184, the host machines 1101, 1108, and 1109 areserving as designated host machines for VXLAN500 for handing trafficfrom PHs 1185 and 1186, and the host machines 1110, 1111, and 1106 areserving as designated host machines for VXLAN500 for handing trafficfrom PHs 1187 and 1188. Though not illustrated, in some embodiments,some of the network segments are inherently distributed so there wouldbe no need for designated instances for handling traffic from physicalhosts of those network segments. For example, in some embodiments, someVXLAN network segments have physical hosts that are capable ofdistributed routing and therefore do not need MPREs in other hostmachines as designated instances.

Each network segment (and the LIF for that network segment) has itsmultiple LIF IP addresses assigned to different host machines. Forexample, the LIF for VLAN200 has three IP addresses 2.1.2.251,2.1.2.252, and 2.2.253, and each of these IP addresses is assigned to adifferent host machine (2.1.2.251 is assigned to the host machine 1104,2.1.2.252 is assigned to the host machine 1105, and 2.1.2.253 isassigned to the host machine 1106). As mentioned earlier by reference toFIG. 10, some host machines serve as designated host machine fordifferent IP addresses from different LIFs/network segments. Asillustrated in FIG. 11, the host machine 1101 (PMAC1) is serving asdesignated host machine (i.e., hosting a designated instance MPRE) forboth 2.1.1.251 of VLAN100 and 2.1.3.251 for VXLAN500. The host machine1106 (PMAC6) is serving as designated host machine for both 2.1.2.253 ofVLAN200 and 2.1.4.253 of VXLAN600.

b. Enabling Ingress ECMP Using Multiple LIF Addresses

As mentioned, in some embodiments, having multiple designated instancesper LIF gives a physical host using that LIF a list of choices whenselecting a next hop. A physical host having such a list is able toselect one designated instance as destination, for say, to balance theload across different designated instances. To provide such a list tothe physical hosts of a particular network segment, some embodimentsadvertise the IP addresses of the LIF of that particular network segmentas a list of available next hops.

FIG. 12 conceptually illustrates the advertising of LIF IP addresses asa list of next hops to physical hosts in the network virtualizationinfrastructure 1100. The network infrastructure 1100 is implementing theLRE 1000, whose LIF for VLAN100 is assigned IP addresses 2.1.1.251,2.1.1.252, and 2.1.1.253. The physical hosts 1181 and 1182 are bothrouters in the VLAN100. As illustrated, the network virtualizationinfrastructure 1100 includes a network controller 1250. The networkcontroller 1250 advertises the LIF IP addresses for VLAN100 as a list ofnext hops 1210 to the physical hosts 1181 and 1182. The physical hostscan resolve these IP addresses into L2 MAC addresses by perform ARPoperations on these LIF IP addresses.

The controller 1250 also selects the host machines to serve as thedesignated instances/designated host machines for those advertised LIFIP addresses. As illustrated, the controller 1250 selects the hostmachine 1101 as the designated host (i.e., its MPRE as the designatedinstance) for the LIF IP address 2.1.1.251, the host machine 1102 as thedesignated host for the LIF IP address 2.1.1.252, and the host machine1103 as the designated host for the LIF IP address 2.1.1.253. When thephysical hosts subsequently request address resolution for theirreceived next hop IP addresses, some embodiments provide the PMACs ofthe selected designated instances/designated hosts as the resolved L2MAC addresses to the requesting physical hosts. Address resolution ofLIF IP addresses will be described further below in Section II.c.

Once a list of designated instances is made available to a physicalhost, the physical host is able to select any one of the designatedinstances as a next hop into the logical network. Such selection can bebased on any number of criteria and can be made for any number ofpurposes. In some embodiments, a physical host selects a designatedinstance as the next hop based on current network traffic information inorder to balance the traffic load between the different designated hostmachines. In some embodiments, a PH uses the list of designatedinstances to perform ECMP (Equal Cost Multi-path Routing) algorithms oningress network traffic to the logical network.

FIG. 13 illustrates a network system 1300 in which routers for ingressnetwork traffic into a logical network perform ECMP based on lists ofadvertised available next-hops. The network system 1300 includes thenetwork virtualization infrastructure 1100, edge routers 1361 and 1362,and core routers 1371-1373. The network virtualization infrastructure1100 is implementing the LRE 1000. The core routers 1371-1373 arerouters of the internet backbone at a client site, and the edge routers1361 and 1362 are gateways for network traffic into the networkvirtualization infrastructure 1100. The edge routers 1361 and 1362 havereceived the list 1210 of IP addresses that it can use as the next hopinto the virtualization infrastructure 1100. The routers 1361 and 1362are network nodes in VLAN100, and the list of IP addresses are the LIFIP addresses for VLAN100.

Each of the core routers 1371-1373 performs ECMP algorithms to selectone of the edge routers 1361-1362 as the next hop for traffic flowingfrom the client site towards the network virtualization infrastructure1100. Each of the edge routers 1361-1362 in turn performs its own ECMPalgorithm to select one of the designated instances as the next hop fortraffic into the network virtualization infrastructure 1100. In someembodiments, at least some of the routers perform the ECMP algorithms inorder to balance the traffic and/or computation load among downstreamrouters. In some embodiments, such an ECMP algorithm is based on dynamicnetwork traffic status, where the selection of the next hop is cognizantof the current traffic load on each of the designated instances. In someembodiments, the ECMP algorithm selects a next hop by blindly hashingthe ingress data packet without regard to any real-time network trafficstatus.

The edge router 1361 has a list 1341 and the edge router 1362 has a list1342. Both the lists 1341 and 1342 are derived from the advertised listof LIF IP addresses 1210 that includes 2.1.1.251, 2.1.1.252, and2.1.1.253. Each of the routers selects a next hop from uses its list ofIP addresses. For example, the edge router 1361 uses its list 1341 toperform ECMP and determines that 2.1.1.252 is a better next hop than2.1.1.251 and 2.1.1.253 for a particular data packet. The edge router1361 then selects 2.1.1.252 as the destination IP. In the example ofFIG. 13, the MPRE running on the host machine 1102 has been selected asthe designated instance for the IP address 2.1.1.252, which has L2address “PMAC2”. The particular data packet destined to the IP address2.1.1.252 will therefore be sent to the host machine 1102 by using theL2 address “PMAC2”.

FIG. 14 conceptually illustrates a process 1400 performed by a physicalhost for selecting a designated instance of an LRE for routing. Theprocess starts when the physical host receives (at 1410) a list of LIFIP addresses as possible next hops. The process then determines (at1420) if it has a packet that needs to be routed by the LRE, i.e., apacket whose destination is in the logical network using the LRE. If so,the process proceeds to 1430. Otherwise, the process 1400 ends.

At 1430, the process updates network information. The process thenselects (at 1440) an IP address as the next hop. Some embodiments selecta next hop based on real time network information update in order toachieve load balancing. Some embodiments do not use such networkinformation update but rather rely on random selection (e.g., simplehashing) to achieve load balancing. Some embodiments use other types ofECMP algorithms for selecting a next hop.

The process next determines (at 1450) whether the selected next hop IPaddress has a corresponding resolved L2 address. The resolved L2 addressis the actual MAC address of the host machine that is chosen as thedesignated host (and hosting the designated LRE instance) for the nexthop IP address. If the selected next hop has a resolved L2 address, theprocess proceeds to 1460 to forward the packet. Otherwise, the processperforms (at 1455) address resolution operation in order to resolve theselected next hop IP address (e.g., by sending ARP request for theselected next hop IP address).

Once the next IP address has been resolved into an L2 address, theprocess forwards (1460) the packet by using the resolved L2 address. Theprocess 1400 then returns to 1420 to see if there is another packet tobe forwarded by the LRE. The resolution of addresses by designatedinstances will be further described in Section II.c below.

FIG. 15 conceptually illustrates a process 1501 for providing multipledesignated instances to external physical host machines. The process1501 creates an LRE and provides a list of LIF IP addresses as next hopsto the external host machines. It also selects host machines to serve asdesignated instances. The process 1501 starts by creating (at 1510) anLRE for a logical network for a particular client. The LRE includesdifferent LIFs for interfacing with different network segments.

The process then assigns (at 1515) a set of IP addresses for a LIF.Next, the process assigns (at 1520) a designated instance to each IPaddress of the LIF. Each designated instance is an MPRE residing on ahost machine. The process then advertises (at 1525) the list of IPaddress for the LIF as a list of available next hops to external hostmachines (e.g., edge routers) connected to that LIF. The process thenrepeats 1515 through 1525 until it determines (at 1530) that all LIFs inthe LRE have a set of IP addresses and a set of corresponding designatedinstances. In some embodiments, each LIF is assigned a unique set of IPaddresses and no two LIFs share a same IP address. In some embodiments,an MPRE of a host machine can serve as the designated instance for twoor more different IP addresses from different LIFs.

Once the designated instances for the LIF IP addresses have been chosen,the process produces (at 1540) a configuration for the LRE. The processthen pushes (1545) the LRE configuration to each of the host machines inthe network virtualization infrastructure. Some of the host machinesreceiving the configuration would learn that it has been chosen as adesignated host machine (i.e., having a designated instance MPRE) andperform the functions of a designated instance. The configuration of anLRE will be described further in Section III below. The process 1501then ends.

FIG. 15 also conceptually illustrates a process 1502 for dynamicallyupdating assignment of designated instances after the list of LIF IPaddresses has already been pushed to the external physical routers. Theprocess 1502 starts when it receives (at 1550) update of traffic loadinformation for a particular network segment. Such information in someembodiments provides status of the network traffic flow in each networksegment as well as the network/computation load on each of the hostmachines. The process then identifies (at 1560) a LIF for a networksegment for which it is necessary to change the assignment of designatedinstances. Next, the process updates (1570) the assignment of designatedinstances for that LIF based on the updated traffic load information.Finally, the process pushes (at 1580) updated configuration to the hostmachines in order to inform the host machines of the new designatedinstance assignment. The process 1502 then ends. The physical hosts canthen perform ARP on their next hop IP address to find out the PMACaddress of the newly selected designated instances.

c. Address Resolution Using Multiple LIF Addresses

The routing operations illustrated in FIGS. 7 and 9 rely on routingtable entries in MPREs for translating L3 IP addresses into L2 MACaddresses. Packets coming from physical hosts (PHs) in particular relyon routing table entries in designated instances for routing. In someembodiments, these entries are filled by address resolution protocols(ARP) initiated by PHs or by DIs themselves. Furthermore, a PH that hasreceived a list of IP addresses as next hops (such as the routers 1181,1182, 1361 and 1362 in FIGS. 12 and 13) also performs ARP operation totranslate the received L3 IP address into L2 MAC addresses in someembodiments. In other words, in order to use the received LIF IPaddresses as next hops, a PH in some embodiments performs ARP in orderto ascertain the PMAC addresses of the designated instances.

For some embodiments, FIG. 16 illustrates ARP operations for resolvingLIF IP addresses advertised to the PHs. The figure illustrates ARPoperations by PHs 1681 and PH 1682. The PHs 1681 and 1682 have eachreceived a list of next hops from a logical network 1600. The PHs 1681and 1682 are both network nodes in a network segment VLAN100, and thelist provides a list of IP address for the LIF of VLAN100, whichincludes 2.1.1.251, 2.1.2.252, and 2.1.1.253. The PH 1681 is maintaininga routing table 1641 and the PH 1682 is maintaining a routing table1642.

The logical network 1600 is implemented over an array of host machines,including host machines 1601 and 1602. The logical network 1600 isimplementing an LRE 1650, and the host machines of the logical network,including the host machines 1601 and 1602, are each running a localinstance of the LRE 1650 as its MPRE. The PMAC address of the hostmachine 1601 is “PMAC1”, and its MPRE has been chosen as the designatedinstance for the LIF address 2.1.1.251. The PMAC address of the hostmachine 1602 is “PMAC2”, and its MPRE has been chosen as the designedinstance for the LIF address 2.1.2.252.

FIG. 16 illustrates the ARP operations by PHs for resolving the LIF IPaddresses in nine operations labeled ‘1’ through ‘9’. At operation ‘1’,the PH 1681 selects the IP address 2.1.1.251 as a next hop, but itsrouting table 1641 does not have an entry for 2.1.1.251. The PH 1681 inturn broadcast an ARP query message for the IP address 2.1.1.251 byusing “ffffffffffff” as destination MAC address. At operation ‘2’, thehost machine 1601 receives the ARP query broadcast. Realizing that it isthe designated instance for the IP address 2.1.1.251, it sends an ARPreply to the PH 1681 indicating that the MAC address for the IPaddresses is “PMAC1”. At operation ‘3’, the PH 1681 receives the ARPreply and updates its routing table entry for 2.1.1.251 with “PMAC1”.

At operation ‘4’, the PH 1681 selects the IP address 2.1.2.252 as a nexthop, but its routing table 1641 does not have an entry for the2.1.2.252. The PH 1681 in turn broadcast an ARP query message for the IPaddress 2.1.2.252. At operation ‘5’, the host machine 1602 receives theARP query broadcast. Realizing that it is the designated instance forthe IP address 2.1.2.252, it sends an ARP reply to the PH 1681indicating that the MAC address for the IP addresses is “PMAC2”. Atoperation ‘6’, the PH 1681 receives the ARP reply and updates itsrouting table entry for 2.1.2.252 with “PMAC2”. After operations ‘1’through ‘6’, the router 1681 will be able to use the MPREs of the hostmachines 1601 and 1602 for routing.

At operation ‘7’, the PH 1682 also selects the IP address 2.1.2.252 as anext hop, but its routing table 1642 does not have an entry for the2.1.2.252. The PH 1682 in turn broadcast an ARP query message for the IPaddress 2.1.2.252. At operation ‘8’, the host machine 1602 receives theARP query broadcast. Realizing that it is the designated instance forthe IP address 2.1.2.252, it sends an ARP reply to the PH 1682indicating that the MAC address for the IP addresses is “PMAC2”. Atoperation ‘9’, the PH 1682 receives the ARP reply and updates itsrouting table entry for 2.1.2.252 with “PMAC2”. After operations ‘7’through ‘9’, router 1682 will be able to use the MPRE of the hostmachine 1602 for routing.

In some embodiments, the designated instances also serve as ARP proxies.In some embodiments, a designated instance performs ARP of its own if itis not able to resolve a destination IP address. FIG. 17 a-b illustratesthe designated instances 1601 and 1602 acting as ARP proxies when theyreceive data packets with unknown destination IP addresses from the PH1681. As illustrated, the PH 1681 has already resolved its next hop LIFIP addresses 2.1.1.251 and 2.1.2.252 into “PMAC1” and “PMAC2” fromprevious ARP operations (i.e., the operations illustrated in FIG. 16).The PH 1681 is therefore able to select either “PMAC1” or “PMAC” forrouting. In some embodiments, such a selection is based on ECMPalgorithm for load balancing purposes as discussed above in SectionII.b.

In operations labeled ‘1’ to ‘12’, FIGS. 14 a-b illustrates the routingof packets 1771 and 1772 to VMs 1721 and 1734 through designatedinstances in host machines 1601 and 1602. At operation ‘1’, the PH 1681sends packet 1771. The packet 1771 has “PMAC1” as its destinationaddress and “2.1.2.101” as its destination IP address. The MAC address“PMAC1” corresponds to the MPRE of the host machine 1601. The PH 1681 atthis operation has selected 2.1.2.101 (PMAC1) over 2.1.3.102 (PMAC2)according to a selection algorithm (e.g., ECMP for load balancing), eventhough both IP addresses of the LIF for VLAN100 has been resolved.

At operation ‘2’, the host machine 1601 receives the packet 1771 basedon the MAC address “PMAC1”, but its routing table 1741 cannot resolvethe IP address 2.1.2.101. At operation ‘3’, the MPRE of the host machine1601 broadcast an ARP query for the destination IP address 2.1.2.101.

At operation ‘4’, the MPRE of a host machine 1701 replies to the ARPquery because the host machine 1701 is hosting a VM 1721, whose IPaddress is 2.1.2.101. The ARP reply indicates that the MAC address for2.1.2.101 is “MAC21”. At operation ‘5’, the host machine 1601 receivesthe ARP reply and updates its routing table 1741 for the entry for2.1.2.101. At operation ‘6’, having resolved the destination IP address2.1.2.101 for the packet 1771, the host machine 1601 sends the datapacket 1771 to the host machine 1701 and to the VM 1721 by using “MAC21”as the destination address.

At operation ‘7’, after sending the packet 1771 to the designatedinstance for 2.1.1.251 (PMAC1), the PH 1681 sends the packet 1772 to thedesignated instance for 2.1.2.252 (PMAC2). The packet 1772 has “PMAC2”as its destination address and “2.1.3.102” as its destination IPaddress. The MAC address “PMAC2” corresponds to the MPRE of the hostmachine 1602. The PH 1681 at this operation has selected 2.1.3.102(PMAC2) over 2.1.2.101 (PMAC1) according to a selection algorithm (e.g.,ECMP for load balancing), even though both IP addresses of the LIF forVLAN100 has been resolved.

At operation ‘8’, the host machine 1602 receives the packet 1772 basedon the MAC address “PMAC2”, but its routing table 1742 cannot resolvethe IP address 2.1.3.102. At operation ‘9’, the MPRE of the host machine1602 broadcast an ARP query for the destination IP address 2.1.3.102. Atoperation ‘10’, the MPRE of a host machine 1703 replies to the ARP querybecause the host machine 1703 is hosting a VM 1734, whose IP address is2.1.3.102. The ARP reply indicates that the MAC address for 2.1.3.102 is“MAC34”. At operation ‘11’, the host machine 1602 receives the ARP replyand updates its routing table 1742 for the entry for 2.1.3.102. Atoperation ‘12’, having resolved the destination IP address 2.1.3.102 forthe packet 1772, the host machine 1602 sends the data packet 1772 to thehost machine 1703 and to the VM 1734 by using “MAC34” as the destinationaddress.

Once the routing table of a designated instance has an MAC addressresolution for a destination IP address, any subsequent data packethaving the same destination IP address can use the resolved MAC addressand would not cause the designated instance to initiate another ARPrequest for that same destination IP address. FIG. 18 illustrates thedesignated instance 1601 using its existing routing table entry to routea data packet 1871 from the other PH 1682 without initiating an ARPoperation. As illustrated, the routing table 1741 of the host machine1601 already has an address resolution entry for 2.1.2.101 as “MAC21”from a previous ARP operation (i.e., the operations illustrated in FIG.17 a-b). In operations labeled ‘1’ to ‘3’, the figure illustrates therouting of the packet 1871 from the PH 1682 to the VM 1721.

At operation ‘1’, the PH 1682 sends the packet 1871. The packet 1871 has“PMAC1” as its destination address and “2.1.2.101” as its destination IPaddress. The MAC address “PMAC1” corresponds to the MPRE of the hostmachine 1601. At operation ‘2’, the host machine 1601 receives thepacket 1871 based on the MAC address “PMAC1”, and its routing table 1741already has an entry for resolving the IP address 2.1.2.101 into“MAC21”. The routing table 1741 also adds an entry based on the packet'ssource IP address and MAC address (i.e., 2.1.2.11 and “MAC11” of the PH1682) for future use. At operation ‘3’, the host machine 1601 sends thedata packet 1871 to the host machine 1701 and to the VM 1721 by using“MAC21” as the destination address.

In some embodiments, the designated instances not only resolve IPaddresses for packets that comes from external PHs, but also for packetscoming from host machines running a local instance of the LRE. FIG. 19illustrates the routing of a packet 1971 from a VM 1745 in a hostmachine 1705 operating a MPRE to a physical host that is not operating aMPRE. The routing utilizes routing table entries in the availabledesignated instances 1601 and 1602 for the LIF VLAN100. The routingtable 1741 of the host machine 1601 already has an entry for 2.1.2.11 as“MAC11” from a previous routing operation (i.e., the routing of thepacket 1871 from the PH 1682 in FIG. 18).

In operations labeled ‘1’ through ‘6’, FIG. 19 illustrates the routingof the packet 1971 from the VM 1745 to the PH 1682. At operation ‘1’, aVM 1745 running on a host machine 1705 is sending the data packet 1971,which has a destination IP 2.1.2.11 and destination MAC address “VMAC”.As mentioned earlier, “VMAC” is the MAC address used by a VM whenaddressing its own local MPRE. Since the MPRE in the host machine 1705is not able to resolve the destination IP address 2.1.2.11, the hostmachine 1705 sends out a request for resolution to the designatedinstances 1601 and 1602 at operations ‘2’ and ‘3’, respectively.

In some embodiments, an MPRE that needs to resolve a destination IPaddress would make a request for address resolution to a designatedinstance. In some embodiments, an MPRE would make such an addressresolution request to a designated instance that is associated with aLIF address that is in same IP subnet as the destination IP address. Inthe example of FIG. 19, the host machine 1602 is a designated instancefor the IP address 2.1.2.252, which is in the same IP subnet asdestination address 2.1.2.11. The MPRE in the host machine 1705therefore makes the address resolution request to the designatedinstance 1602 rather than to 1601, whose IP address 2.1.1.251 is in adifferent IP subnet. In some embodiments, each designated instance isfor resolving IP addresses that are in the same subnet as its assignedLIF IP address.

The host machine 1601 at operation ‘4’ examines its routing table andfound an entry for the IP address 2.1.2.11 as “MAC11” and replies to theMPRE in the host machine 1705 in operation ‘5’. Finally, at operation‘6’, the MPRE of the host machine 1705 sends the data packet 1671 to thePH 1682 by using the MAC address “MAC11”, which is the MAC address ofthe PH 1682.

In some embodiments, the address resolution requests to designatedinstances and address resolution replies from designated instances areUDP messages. In the example of FIG. 19, one of the designated instanceshas a routing table entry for the destination IP address, and wastherefore able to reply to the address resolution request with its ownrouting table entry. In some embodiments, when a designated instance isnot able to resolve a destination IP address upon receiving an addressresolution request, it will perform an ARP operation in order to resolvethe unknown IP address. FIG. 20 illustrates an ARP operation performedby a designated instance when it is unable to resolve an IP address uponreceiving an address resolution request.

FIG. 20 illustrates the routing of a packet 2071 from a VM 1756 to a PH1683, which is also a physical host in VLAN100 and connected to the LREby a LIF for the network segment VLAN100. Its IP address is 2.1.1.12 andits MAC address is “MAC12”. This MAC address is unknown to thedesignated instances 1601 and 1602. In other words, if a designatedinstance receives an address resolution request for the IP address2.1.1.12, it would perform an ARP operation.

In operations labeled ‘1’ through ‘8’, FIG. 20 illustrates the routingof a packet 2071 from the VM 1756 to the PH 1683. At operation ‘1’, theVM 1756 running on a host machine 1706 is sending the data packet 2071,which has a destination IP 2.1.1.12 and destination MAC address “VMAC”.As mentioned earlier, “VMAC” is the MAC address used by a VM whenaddressing its own local MPRE. Since the MPRE in the host machine 1706is not able to resolve the destination IP address 2.1.1.12, it sends outa request for resolution to the designated instances 1601 and 1602 inoperations ‘2’ and ‘3’ respectively. In the example of FIG. 20, the hostmachine 1601 is a designated instance for the IP address 2.1.1.251,which is in the same IP subnet as destination address 2.1.1.12. The MPREin the host machine 1706 therefore makes the address resolution requestto the designated instance 1601 rather than 1602, whose IP address2.1.2.252 is in a different IP subnet.

At operation ‘4’, the host machine (designated instance) 1601 examinesits routing table and realizes that it does not have an entry forresolving IP address 2.1.1.12. It therefore broadcasts an ARP requestfor the IP address 2.1.1.12. At operation ‘5’, the PH 1683, whose IPaddress is 2.1.1.12, replies to the ARP request with its MAC address“MAC12”. At operation ‘6’, the designated instance 1601 receives the ARPreply from the PH 1683, and updates its own routing table 1741. Atoperation ‘7’, the designated instance 1601 sends address resolutionreply message to the MPRE in the host machine 1706, informing the MPREthat the MAC address for the IP address 2.1.1.12 is “MAC12”. Atoperation ‘8’, the MPRE in the host machine 1756 forwards the packet2071 to the PH 1683 by using “MAC12” as the destination MAC address.

In the examples of FIGS. 19 and 20, the packets being routed (1971 and2071) are sourced by VMs operating on host machines that are notdesignated instances (VMs 1745 and 1756 running on host machines 1705and 1706). However, one of ordinary skill would understand that theoperations illustrated in FIGS. 19 and 20 can also be performed for a VMthat is operating on a designated instance host machine.

For some embodiments, FIG. 21 conceptually illustrates a process 2100for processing a data packet at an MPRE. In some embodiments, theprocess 2100 is performed by MPREs that are designated instances as wellas MPREs that are not designated instances. The process 2100 starts whenit receives (at 2105) a packet based on a destination MAC address. Thedestination MAC address can either be a broadcast MAC address (e.g.,ffffffffffff) or the MAC address of the receiving MPRE (i.e., its PMACaddress or the generic VMAC address of all MPREs). The process thendetermines (at 2110) whether the packet is an ARP query for an IPaddress. If the packet is an ARP query, the process proceeds to 2120.Otherwise, the process proceeds to 2140.

At 2120, the process examines if this MPRE is a designated instance ofthe IP address being ARP-queried. If this MPRE is the designatedinstance for the IP address being ARP-queried, the process responds (at2130) to the ARP query with its own unique PMAC address and ends.Otherwise the process 2100 ignores (at 2135) the ARP query and ends.

At 2140, the process determines if the destination IP address is in therouting table of the MPRE. If the destination IP address is not in therouting table, the process proceeds to 2150. If the destination IP is inthe routing table, the process routes (at 2145) the packet by using therouting table entry for the destination IP address to find thecorresponding MAC address. The packet then forwards (at 2148) the packetby using the MAC address as the destination address for the packet. Thisforwarding operation is performed by using the MPSE of the host machinein some embodiments. The process 2100 then ends.

At 2150, the process selects a designated instance for resolving the IPaddress. As mentioned, in some embodiments, each LIF has multiple IPaddresses, and each of the IP addresses is assigned to a designatedinstance. In some embodiments, the process would make the addressresolution request to a designated instance that corresponds to a LIF IPaddress that is in the same IP subnet as the destination IP address. Theprocess then determines (at 2155) if this MPRE is itself the selecteddesignated instance. If this MPRE is the selected designated instanceitself, process proceeds to 2180. If this MPRE is not the selecteddesignated instance, or is not a designated instance at all, the processrequests (at 2160) address resolution from the selected designatedinstance. The process then receives (at 2165) the address resolutionfrom the designated instance. In some embodiments, such addressresolution requests and replies are transmitted as UDP messages betweenthe designated instance and the host machine requesting the addressresolution. The process then updates (at 2170) the routing table of theMPRE based on the received address resolution, and proceeds to 2145 toroute the data packet.

At 2180, the process performs ARP operation to resolve the IP address,since the MPRE is the selected designated instance but cannot resolvedestination IP address from its existing routing table entries. Aftermaking the ARP request and receiving the reply for the ARP, the process2100 proceeds to 2170 to update its routing table, route (at 2145) thedata packet, forwards (at 2148) the data packet, and ends.

For some embodiments, FIG. 22 conceptually illustrates a process 2200for performing address resolution at a designated instance MPRE. Theprocess starts when it receives (at 2210) an address resolution requestmessage from a host machine (e.g., from an MPRE performing the process2100) for a destination IP address with unknown MAC address. The processthen determines (at 2220) if it is able to resolve the requesteddestination IP address locally, i.e., if the requested address is in theMPRE's own routing table. If so, the process proceeds to 2250. If theprocess cannot resolve the requested address, it proceeds to 2230 tobroadcast an ARP request for the requested destination IP address. Theprocess then updates (at 2240) its routing table after it receives thecorresponding ARP reply bearing the resolved MAC address. The processthen replies (at 2250) to the address resolution request by informingthe requester of the resolved MAC address. The process 2200 then ends.In some embodiments, the address resolution request message and theaddress resolution reply messages are UDP messages.

FIG. 23 conceptually illustrates a process 2300 for performing packetrouting and forwarding at an MPRE in some embodiments. In someembodiments, the process 2300 is performed as part of the operations2145 and 2148 in the process 2100. The process 2300 starts when itreceives a data packet with a resolved destination MAC address. The MACaddress can come as a part of the data packet being already resolved atthe sender. This MPRE may also resolve the MAC address locally by usingits own routing table, requesting address resolution from a designatedinstance, or by performing an ARP operation. The resolved MAC addressinforms the process what is the next hop for the message. The processthen determines (at 2320) whether the destination MAC address belongs toa VM running on this host machine. If so, the process forwards (at 2325)the packet to the VM identified by the MAC address. Otherwise theprocess forwards (at 2330) the packet out of this host machine. Theprocess 2300 then ends.

III. Configuration of Logical Routing Element

In some embodiments, the LRE instantiations operating locally in hostmachines as MPREs (either for routing and/or bridging) as describedabove are configured by configuration data sets that are generated by acluster of controllers. The controllers in some embodiments in turngenerate these configuration data sets based on logical networks thatare created and specified by different tenants or users. In someembodiments, a network manager for a network virtualizationinfrastructure allows users to generate different logical networks thatcan be implemented over the network virtualization infrastructure, andthen pushes the parameters of these logical networks to the controllersso the controllers can generate host machine specific configuration datasets, including configuration data for LREs. In some embodiments, thenetwork manager provides instructions to the host machines for fetchingconfiguration data for LREs from the controllers.

For some embodiments, FIG. 24 illustrates a network virtualizationinfrastructure 2400, in which logical network specifications areconverted into configurations for LREs in host machines (to beMPREs/bridges). As illustrated, the network virtualizationinfrastructure 2400 includes a network manager 2410, one or moreclusters of controllers 2420, and host machines 2430 that areinterconnected by a physical network. The host machines 2430 includeshost machines 2431-2439, though host machines 2435-2439 are notillustrated in this figure.

The network manager 2410 provides specifications for one or more usercreated logical networks. In some embodiments, the network managerincludes a suite of applications that let users specify their ownlogical networks that can be virtualized over the network virtualizationinfrastructure 2400. In some embodiments the network manager provides anapplication programming interface (API) for users to specify logicalnetworks in a programming environment. The network manager in turnpushes these created logical networks to the clusters of controllers2420 for implementation at the host machines.

The controller cluster 2420 includes multiple controllers forcontrolling the operations of the host machines 2430 in the networkvirtualization infrastructure 2400. The controller creates configurationdata sets for the host machines based on the logical networks that arecreated by the network managers. The controllers also dynamicallyprovide configuration update and routing information to the hostmachines 2431-2434. In some embodiments, the controllers are organizedin order to provide distributed or resilient control plane architecturein order to ensure that each host machines can still receive updates androutes even if a certain control plane node fails. In some embodiments,at least some of the controllers are virtual machines operating in hostmachines.

The host machines 2430 operate LREs and receive configuration data fromthe controller cluster 2420 for configuring the LREs as MPREs/bridges.Each of the host machines includes a controller agent for retrievingconfiguration data from the cluster of controllers 2420. In someembodiments, each host machine updates its MPRE forwarding tableaccording to a VDR control plane. In some embodiments, the VDR controlplane communicates by using standard route-exchange protocols such asOSPF (open shortest path first) or BGP (border gateway protocol) torouting peers to advertise/determine the best routes.

FIG. 24 also illustrates operations that take place in the networkvirtualization infrastructure 2400 in order to configure the LREs in thehost machines 2430. In operation ‘1’, the network manager 2410communicates instructions to the host machines for fetchingconfiguration for the LREs. In some embodiments, this instructionincludes the address that points to specific locations in the clustersof controllers 2420. In operation ‘2’, the network manager 2410 sendsthe logical network specifications to the controllers in the clusters2420, and the controllers generate configuration data for individualhost machines and LREs.

In operation ‘3’, the controller agents operating in the host machines2430 send requests for LRE configurations from the cluster ofcontrollers 2420, based on the instructions received at operation ‘2’.That is, the controller agents contact the controllers to which they arepointed by the network manager 2410. In operation ‘4’, the clusters ofcontrollers 2420 provide LRE configurations to the host machines inresponse to the requests.

FIG. 25 conceptually illustrates the delivery of configuration data fromthe network manager 2410 to LREs operating in individual host machines2431-2434. As illustrated, the network manager 2410 creates logicalnetworks for different tenants according to user specification. Thenetwork manager delivers the descriptions of the created logicalnetworks 2510 and 2520 to the controllers 2420. The controller 2420 inturn processes the logical network descriptions 2510 and 2520 intoconfiguration data sets 2531-2534 for delivery to individual hostmachines 2431-2434, respectively. In other embodiments, however, thenetwork manager generates these configuration data sets, and thecontrollers are only responsible for the delivery to the host machines.These configuration data sets are in turn used to configure the LREs ofthe different logical networks to operate as MPREs in individual hostmachines.

FIG. 26 illustrates the structure of the configuration data sets thatare delivered to individual host machines. The figure illustrates theconfiguration data sets 2531-2537 for host machines 2431-2439. The hostmachines are operating two LREs 2610 and 2620 for two different tenantsX and Y. The host machines 2431, 2432, 2434, and 2437 are eachconfigured to operate a MPRE of the LRE 2610 (of tenant X), while thehost machines 2432, 2433, 2434, and 2435 are each configured to operatea MPRE of the LRE 2620 (for tenant Y). It is worth noting that differentLREs for different logical networks of different tenants can reside in asame host machine, as discussed above by reference to FIG. 4. In theexample of FIG. 26, the host machine 2432 is operating MPREs for boththe LRE 2610 for tenant X and the LRE 2620 for tenant Y.

The LRE 2610 for tenant X includes LIFs for network segments A, B, andC. The LRE 2620 for tenant Y includes LIFs for network segments D, E,and F. In some embodiments, each logical interface is specific to alogical network, and no logical interface can appear in different LREsfor different tenants.

The configuration data for a host in some embodiments includes its VMAC(which is generic for all hosts), its unique PMAC, and a list of LREsrunning on that host. For example, the configuration data for the host2433 would show that the host 2433 is operating a MPRE for the LRE 2620,while the configuration data for the host 2434 would show that the host2434 is operating MPREs for the LRE 2610 and the LRE 2620. In someembodiments, the MPRE for tenant X and the MPRE for tenant Y of a givenhost machine are both addressable by the same unique PMAC assigned tothe host machine.

The configuration data for an LRE in some embodiments includes a list ofLIFs, a routing/forwarding table, and controller cluster information.The controller cluster information, in some embodiments, informs thehost where to obtain updated control and configuration information. Insome embodiments, the configuration data for an LRE is replicated forall of the LRE's instantiations (i.e., MPREs) across the different hostmachines.

The configuration data for a LIF in some embodiments includes the nameof the logical interface (e.g., a UUID), its set of IP addresses, itsMAC address (i.e., LMAC or VMAC), its MTU (maximum transmission unit),its destination info (e.g., the VNI of the network segment with which itinterfaces), whether it is active or inactive on the particular host,and whether it is a bridge LIF or a routing LIF. The configuration datafor LIF also includes a designated instance criteria field 2650.

In some embodiments, the designated instance criteria is an externalfacing parameters that indicate whether a LRE running on a host as itsMPRE is a designated instance and needs to perform address resolutionfor physical hosts. In some embodiments, such criteria for designatedinstances is a list (e.g., 2650) of the IP address for the LIF and thecorresponding identifiers for the host machines selected to serve as thedesignated instance/designated host machine for those IP addresses. Insome embodiments, a host machine that receives the configuration datadetermines whether it is a designated host machine (i.e., operating aMPRE that is the designated instance) for one of the LIF IP addresses byexamining the list 2650. A host machine (e.g., host 2) knows to operateits MPRE as a designated instance for a particular LIF IP address (e.g.,2.1.2.252) when it sees its own identifier associated with thatparticular LIF IP addresses in the designated instance criteria 2650.

In some embodiments, the LREs are configured or controlled by APIsoperating in the network manager. For example, some embodiments provideAPIs for creating a LRE, deleting an LRE, adding a LIF, and deleting aLIF. In some embodiments, the controllers not only provide staticconfiguration data for configuring the LREs operating in the hostmachines (as MPRE/bridges), but also provide static and/or dynamicrouting information to the local LRE instantiations running as MPREs.Some embodiments provide APIs for updating LIFs (e.g., to update theMTU/MAC/IP information of a LIF), and add or modify route entry for agiven LRE. A routing entry in some embodiments includes information suchas destination IP or subnet mask, next hop information, logicalinterface, metric, route type (neighbor entry or next hop or interface,etc.), route control flags, and actions (such as forward, blackhole,etc.).

Some embodiments dynamically gather and deliver routing information forthe LREs operating as MPREs. FIG. 27 illustrates the gathering and thedelivery of dynamic routing information for LREs. As illustrated, thenetwork virtualization infrastructure 2400 not only includes the clusterof controllers 2420 and host machines 2430, it also includes a hostmachine 2440 that operates a virtual machine (“edge VM”) for gatheringand distributing dynamic routing information. In some embodiments, theedge VM 2440 executes OSPF or BGP protocols and appears as an externalrouter for another LAN or other network. In some embodiments, the edgeVM 2440 learns the network routes from other routers. After validatingthe learned route in its own network segment, the edge VM 2440 sends thelearned routes to the controller clusters 2420. The controller cluster2420 in turn propagates the learned routes to the MPREs in the hostmachines 2430.

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 28 conceptually illustrates an electronic system 2800 with whichsome embodiments of the invention are implemented. The electronic system2800 can be used to execute any of the control, virtualization, oroperating system applications described above. The electronic system2800 may be a computer (e.g., a desktop computer, personal computer,tablet computer, server computer, mainframe, a blade computer etc.),phone, PDA, or any other sort of electronic device. Such an electronicsystem includes various types of computer readable media and interfacesfor various other types of computer readable media. Electronic system2800 includes a bus 2805, processing unit(s) 2810, a system memory 2825,a read-only memory 2830, a permanent storage device 2835, input devices2840, and output devices 2845.

The bus 2805 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 2800. For instance, the bus 2805 communicativelyconnects the processing unit(s) 2810 with the read-only memory 2830, thesystem memory 2825, and the permanent storage device 2835.

From these various memory units, the processing unit(s) 2810 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 2830 stores static data and instructions thatare needed by the processing unit(s) 2810 and other modules of theelectronic system. The permanent storage device 2835, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system2800 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 2835.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 2835, the system memory 2825 is a read-and-write memorydevice. However, unlike storage device 2835, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 2825, the permanent storage device 2835, and/or theread-only memory 2830. From these various memory units, the processingunit(s) 2810 retrieves instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 2805 also connects to the input and output devices 2840 and2845. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 2840 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 2845 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 28, bus 2805 also couples electronic system2800 to a network 2865 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 2800 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 14, 15 a-b, 18, 21, 22, and 23) conceptually illustrateprocesses. The specific operations of these processes may not beperformed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process. Thus, one ofordinary skill in the art would understand that the invention is not tobe limited by the foregoing illustrative details, but rather is to bedefined by the appended claims.

What is claimed is:
 1. A method of operating a logical network over anetwork virtualization infrastructure that comprises a plurality of hostmachines, the method comprising: defining a logical routing element(LRE) for routing data packets between different segments of the logicalnetwork, wherein each host machine operates a set of virtual machines(VMs) and a local instance of the LRE as a managed physical routingelement (MPRE) for routing packets from the set of VMs locally at thehost machine, the LRE comprising a plurality of logical interfaces(LIFs), each LIF for interfacing with a different segment of the logicalnetwork, wherein a particular LIF is addressable by a plurality ofdifferent LIF addresses; assigning each LIF address of the particularLIF to a different host machine in the plurality of host machines; andadvertising the plurality of LIF addresses of the particular LIF as aplurality of next-hops to an external router.
 2. The method of claim 1,wherein at least one of the network segments comprises network nodes ina plurality of different IP subnets.
 3. The method of claim 1, whereinthe plurality of different LIF addresses for addressing the particularLIF comprises addresses in different IP subnets.
 4. The method of claim1, wherein the particular LIF is for interfacing a particular networksegment, wherein the particular network segment is an overlayencapsulation network.
 5. The method of claim 4, wherein the overlayencapsulation network comprises a plurality of different IP subnets. 6.The method of claim 4, wherein the overlay encapsulation network isVXLAN.
 7. A method comprising: connecting a physical router to a logicalnetwork, the logical network comprising a plurality of network segmentsand a logical routing element (LRE) for routing data packets between theplurality of network segments, the LRE comprising a plurality of logicalinterfaces (LIF), each LIF for interfacing a network segment, whereinthe physical router is a network node of a particular network segmentthat interfaces the LRE at a particular LIF but does not operate theLRE; and receiving, at the physical router, a plurality of differentaddresses for addressing the particular LIF; and selecting one addressof the received plurality of different addresses as destination forsending a data packet to the particular LIF.
 8. The method of claim 7,wherein selecting one address of the plurality of different addressescomprises performing an Equal Cost Multi-path Routing (ECMP) algorithm.9. The method of claim 7, wherein the logical network is implementedover a plurality of host machines, wherein each host machine isoperating a local instance of the LRE as a managed physical routingelement (MPRE).
 10. The method of claim 9, wherein the selected addresscorresponds to a particular host machine, wherein said routing of thedata packet is performed by the MPRE in the particular host machine. 11.The method of claim 7, wherein at least one of the network segmentscomprises network nodes in a plurality of different IP subnets.
 12. Themethod of claim 7, wherein the received plurality of addresses foraddressing the particular LIF comprises addresses in different IPsubnets.
 13. The method of claim 7, wherein particular network segmentis an overlay encapsulation network.
 14. A system comprising: aplurality of host machines, each host machine operating a set of virtualmachines (VMs) and an instance of a logical routing element (LRE) forrouting data packets between different network segments of a logicalnetwork, wherein the LRE comprises a plurality of logical interfaces(LIF), each LIF for interfacing a network segment of the logicalnetwork, wherein a particular LIF is associated with a plurality ofnetwork layer addresses, wherein each network layer address of theparticular LIF is assigned to a different host machine; and a physicalrouter that uses the LRE for routing by sending data packets to a hostmachine that is assigned a network layer address of the particular LIF.15. The system of claim 14, wherein the physical router is a networknode in a particular network segment that interfaces the particular LIF.16. The system of claim 14, wherein the physical router balances trafficload between different host machines in the set of host machines byselecting a host machine from the set of host machine as a recipient ofdata packets.
 17. The system of claim 14, wherein the instance of theLRE operating in a host machine is acting as a managed physical routingelement (MPRE) of the host machine, wherein the MPRE routs packet fromVMs of the host machine locally at the host machine.
 18. The system ofclaim 14, wherein at least one of the network segments comprises networknodes in a plurality of different IP subnets.
 19. The system of claim18, wherein the at least one network segment is an overlay encapsulationnetwork.
 20. The system of claim 14, wherein the plurality of networklayer addresses of the particular LIF includes IP addresses in differentIP subnets.
 21. A computing device serving as a host machine in alogical network, the computing device executing a computer programcomprising sets of instructions for: executing a plurality of virtualmachines that are in a plurality of different segments of the logicalnetwork; operating an instance of a logical routing element (LRE) as anmanaged physical routing element (MPRE) for routing data packets betweendifferent segments of the logical network, the LRE comprising aplurality of logical interfaces, each logical interface for receivingdata packets from a different segments of the logical network, wherein aparticular logical interface is associated with a plurality of networklayer addresses; and making the MPRE addressable by one of network layeraddresses associated with the particular logical interface of the LRE.22. The computing device of claim 21, wherein the computer programfurther comprising a set of instructions for receiving the plurality ofnetwork layer addresses.
 23. The computing device of claim 22, whereinthe computer program further comprising a set of instructions fordetermining whether the MPRE is a designated instance for one of thereceived plurality of network layer addresses.